The evolution of the industry towards open systems has occurred in stages. The current stage has compute power distributed to all points of the network infrastructure. The issue that LSC was formed to deal with is the next "wave" of this evolution, where the data requirements of the network exceed its ability to handle them. Widely distributed networks make this job even more difficult.
Data is the most important asset of any enterprise. Until adequate measures are adopted to manage it effectively, those assets are at risk.
However, beyond the continuing enhancements in available hardware there are other equally vital issues, including file accessibility, long-term storage (archiving), control of sensitive corporate data, version control and recovery from disasters.
Strategic Research Corporation (formerly Peripheral Strategies), a Santa Barbara, California-based market research firm, projects that active data storage on the average network is growing at 60% per year, and is expected to top 41GB by 1997 (EDGE: Work-group Computing Report, April 1994, v5n204 p6(1)). The increase in storage needs can be attributed to several factors. Applications such as imaging and CAD/CAM, as well as graphical user interfaces, produce files that can average several megabytes each. In the corporate environment, data is migrating from the mainframe to the personal computer-based network. As LAN fault tolerance increases, the data migration is likely to accelerate. In addition, the cost for on-line storage continues to fall, leading more companies to keep more data on-line.
What's missing in most networks are the tools to effectively manage all that data.
As storage requirements increase, it is important to look at how those procedures might be improved. For instance, some data is vital to the organization and must be kept for a long period. Other data is important, but short-lived. Some data is generated as a "by-product" of another process and is not useful after its immediate creation. Some is accessed frequently, some rarely, some never.
If all the data in the enterprise is backed up every day, without regard to its status, it may reflect significant waste and potential flaws in a storage management procedure. For example, data which does not change need not be backed up over and over. This is not to suggest that important data should not be safeguarded, but that significant savings might be achieved by removing it from the daily backup process. Determining which data fits this description is a useful exercise in analyzing the data "profile" of an organization. Removing data from the backup process while leaving it accessible to users is a key feature of automated archiving systems.
A well-designed storage management system provides the mechanisms to implement a disaster control system. The term disaster control is used to include both disaster planning and disaster recovery, which is much different than backup and recovery. Conventional backups are not suitable for a large-scale disaster recovery system.
The high cost of magnetic disk is often used as justification for the need for a more extensive storage management system. There is another reason which is more compelling. In storing enormous amounts of data on magnetic disk, a disk failure is catastrophic considering the value of that data. The selected storage management system should reduce the need for magnetic disk - greatly minimizing that risk.
Organizations are looking at the desirability of keeping data on- or near-line. As these on-line archives grow, and as data addition rates increase, organizations are looking more negatively at the economics of off-line storage. Cost components consist of labor, media, and maintenance of both the file system and the physical storage device. Although up-front costs are sometimes high, on- or near-line storage can drastically reduce the cost of supporting data archives.
The problem is that computer systems have not been designed to manage these tremendous volumes of data. During the first three computing "waves," the economics of keeping large amounts of data on- or near-line were not feasible. Mainframe data managers were designed to keep tight control on large amounts of mostly off-line data. The new higher magnitudes of generated data are creating the need for new paradigms. In some instances, backing up a system can be done with well-established products and procedures. However, if the volume of data makes it physically impossible to do a backup within a certain timeframe, some radical changes are required.
Naturally, this radical change will require time to become "standard operating procedure" and there are various products which vie to become the solution. The time has come for many organizations to make a choice - there is too much at stake (such as the survival of the enterprise) to delay longer. The problem of providing safe storage for vast amounts of data and allowing fast access to it is the main focus of a specialized storage server and its associated storage management system.
Conventional off-line methods are adequate if the need to "re- process" data is small. As this need grows, more care must be taken to select storage media.
The data storage explosion raises a number of important data integrity issues. First, there is much more data now than there was. Second, "cross-platform compatibility" allows much of the data to be "re-used." And third, the data is not sitting idle in a vault. It is being accessed and shuffled in and out of drives.
The first issue requires an archival system to be high-capacity. The second requires easy access and robust error recovery, disaster recovery, and version control. The third requires a media with a long life and durable construction. And since the value of the data dwarfs the value of the hardware, it is vital that the data be accurately recorded in a manner consistent with national and industry standards to be accessible later.
It should be stated that fast access to data cannot easily be expressed as a single number, such as MIPS or FLOPS. This is because a storage system includes processing user requests as well as retrieving data from on-line, near-line, and off-line sources. The fastest processor in the world will not be efficiently utilized if the data is on a slow storage device, or the media on which it resides is not mounted.
Fast access to data must be balanced with cost, longevity and security needs of the organization. This again suggests separating data which needs "instant" access from data which is accessed rarely.
The subject of media choices could fill multiple volumes - however, it is sufficient to say that the variety of data and the varying requirements in terms of access, life span and cost will necessitate close scrutiny to the choice of media and the method of recording data.
File System Limitations
The UNIX operating system cannot spread a file system across multiple media. This means that for each disk or other storage device there is a complete UNIX file system ("mount point"), and administration tasks must be performed on each one individually. This limitation is especially critical when you consider the cataloging needs of an archive system. The maximum file system size is limited based on fixed allocation of inodes. This may result in not effectively partitioning and managing the media, If the capacity of that media exceeds the maximum file system size.
Disk Storage Limitations
UNIX file allocation schemes based on a single disk allocation unit (DAU) size are never ideal. Although selectable by the administrator at initialization of the disk, compromises must be made -- fast access or efficient utilization of space. And UNIX provides no way to ensure contiguous writing of files, critical for fast access times.
Error Recovery
A proper shutdown of the UNIX system ensures that all system
inodes, buffers and superblocks have been flushed to the disk. An
unexpected shutdown can leave the file system in a state of
disarray that can cause a long delay in re-starting the system.
When the system is re-started, there is no guarantee that all files
will be restorable.
As alluded above, "Unexpected events" can cripple a UNIX system. If a system were to grow to a very large number of files a sudden power interruption could be catastrophic.
Disasters
The need for disaster planning is not contested. As the data
capacity grows, the need for disaster planning grows with it.
UNIX has the basic inability to manage removable media. This
means that it alone cannot provide the mechanism for disaster
recovery.
The IDS/SAM system provides this automated archiving environment.
There are many aspects to this issue which not only hinge on future needs, but on those of today. A prominent industry analyst recently wrote, "People managing local-area-networks today are re-learning the data center of 25-30 years ago." Corporations downsizing from mainframes to LANs are finding that the tools they relied on in the past are not available on the LAN. These tools controlled security, availability, data integrity, and ultimately cost control. Business-critical applications and large volumes of data are forcing LAN managers to look for "mainframe-class" storage management capability. These are provided by IDS/SAM.
The logical hierarchical structure of the UNIX file system allows vendors to provide new capabilities without straying from the vfs standard.
By utilizing the vfs interface, any file system can become available to the system, and is not dependent on the version(s) of UNIX on the network.
The SAM-FS unique features are designed to meet the challenging requirements of managing very large quanities of data over long periods of time in a heterogeneous networking and storage environment.
Because archiving systems deal with large amounts of data, SAM-FS is designed to recover from a disaster quickly and efficiently. SAM-FS can help prevent disaster situations and minimize the impact when a disaster does occur.
Recovery from Disk Failure
Should a disk fail, SAM-FS is capable of recovering all data belonging to archived files. If an archive copy exists the disk space assigned to the file is released, leaving the data restorable from the achive image.
Multiple Copies - Support of Multiple Media Types
SAM-FS software's ability to write multiple copies simultaneously allows valuable information to be stored both on-site and off-site, providing additional protection against disasters. It can use disk array technology to ensure that no data will be lost during power interruptions or system fluctuations. SAM-FS supports tape, WORM and rewritable optical devices, enabling data to be stored on media that is appropriate for the required life of that data.
Lights Out Operation
SAM-FS provides features that adhere to the procedures defined by the disaster recovery plan, such as automatically routing data to the specified media, making multiple removable copies of the data (if required), and recording all file information necessary for recovering the data from the media.
Control Structure Dumps
Since the archiving system is managing a large number of files,
directory and file location information is vital to the continued
operation of the system. Regular "dumps" of such data must be
made to removable media to allow a
"hot" or "warm" site to
restore the file system to its last known state in the event of a
disaster.
Data Format - Media Labeling
A workable disaster recovery plan requires that archived information be self-recoverable, even when the system that created the information is unavailable. SAM software records the complete file information in industry standard tar (Tape ARchive) format on all removable media. This enables files to be recovered by any system with a compatible media device, with or without SAM software installed.
SAM provides the flexibility and control needed to implement a disaster recovery plan which can be customized to the particular site's requirements.
During the first three or four decades of computing, proprietary systems reigned. This was primarily due to the lack of common standards and the resulting lack of broadly applicable software systems. Since the personal computer revolutionized computing during the 1980's, customers are demanding that systems from multiple vendors cooperate on enterprise-wide networks. This has required a continuing evolution in the computing industry. Until these changes occurred, open systems were not possible to implement.
Of course, the promise of open systems makes that effort worthwhile. Once the decision has been made to produce an "open" product, each development decision should be made with this overall goal in mind.
"Shortcuts" are often taken by manufacturers to reduce development time, to conform to a proprietary system, or simply from lack of commitment. These shortcuts usually mean that the resulting system is not completely open. A truly open implementation will work with any "flavor" of UNIX, and also any system based on open systems standards.
In many applications the life span of the data will most often exceed the life span of the equipment it is stored upon - including the IDS. Therefore, the primary design mission is to insure that the data can be read by any UNIX system (or any system capable of reading a standard recording format). Data must be portable and not be dependent upon a specific vendor for reading..
Truly open systems mean that the applications, networks, and hardware are virtually transparent to the user - the way it should be. The mechanics of computing should be transparent, to allow users to focus on the data - the most important asset in any enterprise. Thus the promise of open systems is that it provides data portability. Storing the data in a proprietary format or in a file system tied to a particular architecture or vendor is risky and inconsistent with the spirit of open systems.
As technologies come and go, the promise of open systems lies in the portability and security of the data, regardless of platform or operating environment.
The IDS and SAM-FS are designed to support open systems, incorporate standard technology to protect customer investment, and provide broad connectivity and media interchange with other environments. Some IDS/SAM features ae listed below:
Sun Solaris 2 host operating system; Sun SPARC processor technology.
All media is ANSI-labeled. Thus, all SAM-FS written media can be read on other systems including non-UNIX systems, and all externally-written media can be read by SAM-FS.
All archive images ae written in tar format, preserving not only data but also the original path name, ownership and access control.
Complete transparency for all file operations (i.e. open, read, write, close, etc.) whether on-line magnetic disk files, off-line archive files, or removable media.
Support of standard network protocols and applications (i.e. FTP, NFS, rcp, etc.) and the use of off-the-shelf standard network interfaces (i.e. Ethernet, FastEthernet, FDDI, ATM).
Standard off-the-shelf peripheral technology based on SCSI-2 interfaces.
Return to LSC's Home Page.
(c)1994 LSC, INC ALL RIGHTS RESERVED
Integrated Data Server (IDS), Storage and Archiving
Manager (SAM-FS) and Fast File Recovery System are
trademarks of LSC, Inc. All other trademarks are the property of
their respective owners.